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ABSTRACT 

This speech analyzes the critical ingredients of an 
ob jectives-based management system for (1) goal 

determination — decisions regarding the direction of the educational 
enterprise; (2) progress monitoring — the formative evaluation of the 
system's enroute success in attaining its goals; and (3) output 
appraisal — the summative evaluation of the system's effectiveness. 
Several empirical studies, conducted by UCLA and the Instructional 
Objectives Exchange, are reported insofar as they bear on these three 
phases of large scale educational management. (Author) 
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A presentation in a symposium, Perspectives on 
Management Systems Procedures in Education, given 
at the annual meeting of the American Educational 
Research Association, April 3-8, 1972, Chicago, 
Illinois. 
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an apparent defect in human nature which disinclines 
us to subject any enterprise to careful scrutiny until we sense 
it is in some way defective. Without debating whether this failing 
stems from original sin or is merely an acquired shortcoming, there 
is little doubt that we are currently witnessing the results of 
this tendency in the field of education. 

American citizens in increasing numbers have become disenchan- 
ted with the quality of our educational system, and the magnitude 
of this disenchantment has now passed the critical point, so that 
rhetoric no longer satisfies and corrective action is being demanded. 
The problem facing us now is easier to articulate than to answer, 
namely, M How should we go about promoting improvements in the educa- 
tional enterprise?" 
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Systems Analysis Strategies 

Some educators are turning to systems analysis methodology as 
a possible source for satisfactory answers to this perplexing ques- 
tion. For certain of these systems analysis proponents one senses 
an almost religious devotion to their methodology, a devotion in 
which the litany of imput analysis, output analysis, and servoloop 
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feedback must be chanted daily — or at least in every published 
article and speech. 

For me, however, systems analysis approaches derive their merits 
not because they border on the occult but, rather, because they 
reflect a rational attempt to illuminate the arenas in which we must 
make educational decisions. If most people are left to their own 
devices when they must make decisions, they will usually find that 
erroneous perceptions of reality and unconcious biases render those 
decisions less than satisfactory. Surely there are many wise human 
beings who will reach enlightened decisions which all of us would 
applaud, but there are many others who do not operate as meritor- 
iously. If the decisions affect only themselves, we are not all 
that upset if the wrong choice is made* after all, an individual 
pretty well has the right to muck up his own life if he wishes. 

But in the field of education we see that imprudent decisions can 
penalize thousands of students, thus we cannot remain as sanguine 
regarding intuitively based decision-making. Therefore we find an 
increasing number of people, both educators and non— educators, advo- 
cating the use of more formal mechanisms for making decisions regar- 
ding large scale educational enterprises. Customarily, these 
mechanisms have taken a form which more or less resembles a systems 
analysis approach. 

The distinguishing feature of a systems analysis strategy is 
implied by its name. Clearly, there is an attempt to analyze a 
system of some sort, in this instance an educational system. But 
equally critical is the implication that this analysis will be a 
systematic one. Indeed, many people are enamoured of systems 
analysis approaches for precisely that reason, i.e., they tend to 
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reduce tli© capricious decison making which is so characteristic of 
most human endeavors. 

There is another dimension characteristically associated with 
systems analysis approaches which should be noticed, namely, a 
reliance on evidence of the system's effects. This orientation is 
in contrast to alternative approaches which, although systematic and 
analytic, are not essentially empirical methodologies. For instance, 
analytic philosopical approaches are generally not considered to 
be systems analysis strategies even though they may epitomize 
rigorous analysis. 

Large Scale Educational Systems 

This discussion will be restricted to the consideration of 
large scale educational enterprises such as a state school system 
or a large school district. For example, California legislators are 
currently undertaking a serious appraisal of the state s master 
plan for higher education. One of the considerations of the planners 
relates to the development of an evaluation system which will permit 
the state legislators and other concerned citizens within the state 
to judge the quality of the California higher education system. The 
remarks in the following paper will pertain to the management of such 
an evaluation system. 

This does not suggest that the following observations are inap- 
propriate for small systems such as a moderate sized school district 
or even a single school. Yet, in general, the focus will be on the 
recommendations for systems of sufficient magnitude to warrant the 
considerable investment in carrying out the procedures which will be 
described. 
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Obiectives-Based Systems Analysis 



In most systems analysis models there are three sets of questions 
to be answered. These questions are associated with the three 
major phases of managing a system, as follows: 




A Simplified System Model 

There are questions regarding which inputs should be made to the 
system, that is, the purposes for which the system exists and the 
types of resources which will be used to attain those purposes. A 
second set of questions is associated with the actual operati on of 
the system, that is, how well are things working? A final group 
of questions stem from an appraisal of the output of the system, 
that is, was the effectiveness of the system such that it should 
remain essentially unmodified or do v/e have to make some changes 
in it? 

Now there is nothing inherent in systems analysis models which 
require one to employ instructional objectives as an organizing rubric 
in the implementation of a model. There may be preferable classifi- 
cation schemes for organizing the data vhich must be processed in a 
systems analysis scheme. The present paper, however, will be addres- 
sed to a systems analysis strategy in which instructional objectives 
play a prominent role. The choice to employ objectives as the organ- 
izing dimension stems from a belief that statements of instructional 
objectives can serve as a parsimoneous vehicle for communicating 
the information which must be considered at various points in anal- 







yzing the system. Note, for instance, that those individuals oper- 
ating the National Assessment of Educational Progess, surely dealing 
with a large scale educational enterprise, have chosen to employ 
statements of instructional objectives as their organizing rubric. 

For example, a learner's status in connection with an educational 
system may be represented by his or her performance on an examina- 
tion of some sort. Rather than requiring a decision maker to scrut- 

i 

inize the entire examination, we may convey an idea of what the 
examination entails by identifying the learner competencies it was 
designed to measure. Often these competencies can be described as 
a desired status for the learner, hence the equivalent of an instruc- 
tional objective. In addition, many educators are quite familiar 
with the general concept of instructional objectives, this topic 
having received ample attention during the past decade. 

To reiterate, it is not requisite to employ instructional 
objectives as the organizing theme for an educational systems anal- 
ysis model. Nonetheless, the use of objectives for this purpose 
seems to offer some advantages and, accordingly, the remainder of 
this paper will describe a systems analysis model for large educa- 
tional enterprises which prominently employs instructional objec- 
tives. 

Measurable Objectives 

It is important to note at the outset of this discussion that, 
unless otherwise noted, we will employ the phrase "instructional 
objective" to represent a measurable instructional objective. Because 
measurable objectives communicate one's instructional intentions with 
less ambiguity than broad, general objectives, it would seem particu- 
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larly important to use such objectives in a rational decision making 
scheme where any extra system noise (such as ambiguious symbology) 
will reduce the quality of the decisions. In recent years, of 
course* there have been numerous treatises written regarding techni- 
cal questions of how such measurable objectives should be optimally 
formulated. 

Returning now to the general questions which an educational 
systems analyst must attempt to answer* we can turn first to what is 
perhaps the most important question facing any systems designer.. 

This question is, "To what ends should the system be committed?" 
Putting it another way* "Why should the education system be there in 
the first place?" For an objectives-based systems analysis approach, 
this essentially becomes the problem of goal determination. 

Goal Determination 

In general, the proponent of systems analysis approaches sub- 
scribes to a classic means/ends paradigm. It is anticipated that 
if proper ends can be identified it will be worth the trouble to 
test the efficacy of alternative means to achieve those ends until 
certain means can be identified which do the job. In the field of 
education we are becomming increasingly more sophisticated in 
designing instructional sequences. It thus becomes increasingly 
imperative to identify the most defensible goals of our educational 
systems so that improved instructional means can be directed toward 

the paper ends. 

•phg^-g are at least two approaches to specifying the appropriate 
objectives for a large scale educational system* and these are some- 
what analogous to an inductive versus a deductive approach. Charac- 
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teristically, we have employed an inductive strategy over the 
years in education. According to this scheme the educational planner 
consults various groups with a series of general questions such as 
"What do you want our schools to accomplish?" People typically 
respond to such questions with varying degrees of specificity, so 
usually up to the educational planner to synthesize their 
somewhat diffuse reactions and translate them into more or less 
definitive goal statements. Ralph Tyler's curriculum model which 
has, at least at a theoretical level, been quite influential during 
the past several decades represents such an approach. 

An alternative attack upon the goal determination problem has 
become available in recent years through the establishment of large 
pools of measurable instructional objectives. Various clienteles 
can rate objectives from these pools as to the appropriateness of 
their inclusion in the curriculum of a given education system. In 
this latter approach, therefore, we do not try to derive statements 
of objectives from the value preferences and informal assertions 
of people; rather, we present people with objectives from which 
they choose those they consider most important. 

Perhaps because the latter approach seems to offer a greater 
possibility of systematization through technical refinement, it has 
received more attention during recent months. Particularly as a 
consequence of the needs assessment operations required by federal 
ESEA Title III funding programs, we find more and more educators who 
are attempting to rigorously establish objectives for large scale 
educational endeavors. A number of these efforts have involved the 
use of deductively designated educational objectives. 
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The general strategy in an objectives-based goal determination 
operation involves presentation of alternative sets of educational 
objectives to groups who have a stake in deciding what the goals of 
the system ought to be. These groups then rate, rank, or in other 
ways display their preferences regarding those objectives. The 
expressed preferences of the various groups are then surveyed by 
those who must ultimately decide on the systems' goals and, hopefully, 
more enlightened judments regarding what the system's goals ought to 
be can be made on the basis of such preference data. 

The somewhat new feature of this approach to goal determination 
involves the use of measurable objectives. In previous efforts to 
employ this general strategy educators often used loose, nonmeasur- 
able goals which almost served as Rorschach ink blots for those 
expressing their preferences, that is, people read into nebulous 
goal statements almost anything they wished. As a consequence, it 
was extr em ely difficult to make reasonable contrasts among the 
preferences of various groups. With the use of measurable objectives, 
fortunately, ambiguity is reduced, and as a consequence differences 
among various clienteles are more directly a function of their real 
differences in values rather than confusion regarding the meaning 
of certain goal statements. 

The kinds of groups which might be involved, of course, will 
vary from one educational enterprise to another. For instance, in 
the California higher education evaluation system it would seem 
imperative to involve student groups from the various types of higher 
education institutions within the state, that is, community collegees, 
state colleges, and universities. It would seem equally important to 
involve citizen groups of various kinds, e.g. , parents, businessmen, 
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and other public spirited citizens. This would be an ideal oppor- 
tunity, for example, to secure preference inputs from ethnic and 
other minority groups who often feel large educational systems are 
unresponsive to their particular curricular preferences. It might 
be particularly appropriate to secure the reactions of a group of 
specially designated futurists whose charge would be to consider 
higher education objectives in light of their suitability for the 
1980's and 1990' s, not merely for the next few years. The prefer- 
ences of these groups can be coalesced and represented in straight- 
forward • numerical form in any one of several methods . * 

To illustrate an alternative, somewhat less quantitative method 
of establishing priorities among competing objectives, Professor 
Robert E. Stake of the University of Illinois has recently devised 
an approach to priorities planning in which the decision makers 
consider data such as the preferences of various clienteles. How- 
ever, they also survey the system's requisite resource allocations, 
the payoff probabilities of various objectives, and the relevant 
contingency conditions, that is, circumstances which call for change 
in instructional procedures.** 

Having established the preferred objectives of a system, an 
important second step in the goal determination is to discover the 
degree to which the target learners can already display the hoped 
for behaviors designated by the various reference groups. This is 
where measurable instructional objectives offer considerable 

*See, for example, "Determining Defensible Goals via Educational 
Needs Assessment," Vimcet Associates, 1971, P.O. Box 24714, Los 
Angeles, California 90024. 

**Stake, Robert E. Priorities Planning : Judging the Importance of 
Alternative Objectives , Instructional Objectives Exchange, 1972. 
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advantages, for since the objectives which have been rated by the 
various groups are stated in explicit 'and measurable terms, it is 
a relatively straightforward task to devise measuring devices from 
those objectives and, as a consequence, to measure the learner's 
status. 

Now we would certainly employ item and person sampling tech- 
niques in this approach in order to conserve testing time and to 
make the task economically feasible. But by using such techniques, 
whereby only certain students are sampled and those sampled completed 
only segments of the total measuring devices, we can certainly 
measure student attainment of the high priority goals established 
via the previous rating/ranking procedure. 

The next step, then, is to contrast the learners' status with 
the high priority goals and decide which of these we wish to direct 
our educational enterprise toward. Clearly, there are still a 
great many value judgments to be made at this point, but the hope 
is that by making the decisions as heavily data— based as possible, 
superior decisions will be made. 

This system of goal determination by having different clienteles 
rate extant objectives has been tried out for the past two years by 
the Instructional Objectives Exchange (IOX) with some interesting 
results.* Working with several school districts in Southern Calif- 
ornia, IOX has secured a number of ratings of different objectives 
within the IOX collections by students, teachers, parents, and 
futurists. Various modifications in the directions to the rater 

*Popham, W. James Providing Wide Ranging , Diversely Organized Pool s 
of Instructional Objectives and Measures , Final Report for the U.S. 
Office of Education (Project No. 14-0563, Grant No. OEG-0-9-140563- 
4635(085), Department of Health, Education, and Welfare, 1971. 
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groups have yielded some insights regarding the suitability of such 
matters as the inclusion of sample measurement items to further clarify 
the meaning of the objective, the use of rankings versus ratings, etc. 
Perhaps the most important conclusion drawn from these studies relates 
to the number of objectives to be rated. It now appears to the IOX 
staff, in contrast to earlier expectations, that it is simply unwife 
to present a vast array of instructional objectives to rater groups. 

In other words, even though in certain of the collections of objec- 
tives currently distributed by the Instructional Objectives Exchange 
there are upwards of one or two hundred objectives, it makes little 
sense to attempt to secure ratings of so many objectives. Frankly, 
the educational decision maker generally is not able to process the 
resulting data from such rating analyses. Instead, the conclusion 
from the IOX work to date suggests that it would be far better to 
present a more moderate number of content general objectives, that 
is, measurable objectives which describe a class of learner behav- 
iors rather than a single series of specific test items, and have 
these more general objectives rated by appropriate groups. 

This is a particularly difficult procedure because one of the 
more persistent problems having to do with the technology of objec- 
tives concerns the level of generality of those objectives. Indiv- 
iduals working with instructional objectives since Ralph Tyler s 
early efforts have pointed out that the level of specificity ques- 
tion or, conversely, the level of generality issue, is one of «-he 
most important questions to be resolved in the field of instruc- 
tional objectives. Yet no one seems to have up with a very satis- 
factory solution. Our general estimate at this point, however, is 
that we must find some way to present larger "chunks ' 1 of educational 
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objectives to various groups for ratings. Ending up with more 
than twenty-five or thirty objectives which must be rated presents, 
to most humans, an unmanagable intellectual task. 

It is interesting to note that in certain relatively large 
school districts in the State of California the number of reading 
objectives has been reduced to only three or four, thus a single 
objective, e.g., a student's ability to decode twenty-five words 
randomly drawn from a list of 500, serves to represent the bulk of 

that district's reading effort. 

It is quite apparent that more attention must be given to 
the matter of how general an objective should be in order for it to 
prove servicable in this type of situation. At one extreme, however, 
the experience at IOX to date suggests that far fewer objectives be 
employed than we have characteristically been using. 

Progress Monitoring 

The second set of questions of concern to those involved in 
decisions regarding educational systems concerns the appraisal of 
the progress of the system toward its stipulated goals. One rela- 
tively straightforward method of discerning the degree to which the 
system's goals are being accomplished involves the administration of 
criterion-referenced tests associated with the various system goals 
so that indications of learner progress toward those goals can be 
secured. Goals which, according to measured learner progress, are 
not being achieved can be attacked with alternative instructional 
strategies, additional resources, etc. Goals which are being achieved 
on schedule permit the inference that the instructional program is 
working as well as needed. It is even conceivable, of course, that 
some goals will be achieved ahead of schedule, thereby permitting 
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a shifting of resources toward less effective instructional activities 
It is particularJ.y important in assessing progress toward system 
goals to gather the requisite data as economically as possible. Once 
more, matrix sampling (a technical term for item sampling combined 
with person sampling) is a valuable ally of the educational evaluator. 
It is even possible, when resources are short, to combine matrix 
sampling with goal sampling via a technique whereby progress toward 
only certain goals is monitored. Such an approach, of course, does 
not give .information about all of the system's goals, hence, suffers 
from the defect of supplying only partial information. Nonetheless, 
where a system simply does not have the financial resources to inves- 
tigate satisfactorily the progress toward all goals, the use of goal 
sampling may represent a reasonable alternative. 

It is at this juncture that the evaluator should be particularly 
attentive to unanticipated consequences of the educational system's 
operation. Whereas educational designers can spell out carefully 
the hoped-for outcomes of an educational enterprise, it is often 
the case that some unintended and aversive consequences ensue which 
were simply unforseen by the instructional designers. Michael Scriven 
in a recent paper, argues cogently for the use of goal free evaluation 
in which the educational evaluator is attentive to the consequences 
of an educational system, not to the rhetoric of the instructional 
designers. Scriven suggests that under certain circumstances it may 
be more judicious to examine what happens as a consequence of the 
program rather than what the instructional designers say will occur 
via their stated objectives. Whether one uses goal free evaluation 
approaches or simply employs considerable circumspection regarding 
*Scriven, Michael, Goal Free Evaluation , NIE working paper, 1972. 
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what might have gone wrong, it is quite important to attend to all 
of the effects of a program, not only those which weie intended. 

Output Appraisal 

The final set of questions regarding the management of a large 
scale educational system concerns the final quality of its results. 
Once more, since we are using objectives as an organizing rubric, 
it is possible to develop criterion-referenced tests which are 
congruent with those objectives and administer them on a matrix 
sampling basis to the learners served by the system. Results on 
such measures, combined with measures of unanticipated consequences 
akin to those described in the previous paragraph, will yield the 
kind of evidence necessary to reach a judgment regarding the quality 
of the educational enterprise. 

A particularly thorny problem arises related to the manner in 
which results of such an analysis should be reported. Educational 
evaluators are only now beginning to wrestle seriously with alterna- 
tive vehicles for reporting evaluative results in such a way that 
decision makers can take appropriate action based on the evaluation 
data. All too frequently we find evaluation endeavors resulting in 
encyclopedic final reports which only a person of great devotion 
has the patience to read completely. Brevity is a desirable criterion 
in reporting results of evaluation, studies irrespective of the type 
of evaluation report involved. 

Again, the organizing theme for evaluation, at least for the 
approach being described here, would be the use of instructional 
objectives. Progress toward the major instructional objectives adop- 
ted by the educational system would be reported to the appropriate 

decision making groups, e.g., local school boards, state school 
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boards i university regents, etc. In a general before— after model it 
is important to present the data in as succinct a fashion as possible 
so that those utilizing the results can make more sense out of them. 
Here is where the educational evaluator will have to be particularly 
judicious in the data he selects to report and the manner in which 
those data are described. 

Theory and Practice 

In the foregoing paragraphs a general strategy for the manage- 
ment of large educational enterprises has been described. Clearly, 
the discussion has been at a very general, not a nuts and bolts level. 
Sometimes one has the feeling that in propounding a given theoretical 
point of view an effort to implement it in practice will result in 
chaos. The technical problems are seen as too serious to work out. 

For example, I often hear an aptitude x treatment interaction 
specialist suggest that if we could only sort out pupils' learning 
styles and judiciously mesh them with relevant instructional treat- 
ments, then educational Valhalla would be upon us. At a theoretical 
level I can applaud such a stance, but in my pragmatic heart of 
hearts I really doubt that this can be pulled off, at least in my 
lifetime. The practical problems are too sticky. 

With present propositions, however, this does not seem to be 
the case. Surely there will be technical, procedural questions which 
must be dealt with. For example, exactly which groups will be in- 
volved in rating the objectives that will result in the selection of 
goals for the system? How many learners should be tested in order 
to yield reasonable estimates regarding the progress of the system 
as well as its final output? What kinds of departures from antici- 
pated progress should dictate modifications in the system? These 
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and other problems Ccin be faced and# I believe, resolved by indiv- 
iduals wishing to seriously monitor the progress of a large scale 
educational enterprise. 

But that kind of operation takes more money than most educators 
have been willing to spend. Other than the recently initiated 
Experimental Schools Program of the U.S. Office of Education we see 
few large scale educational enterprises in which ample funds have 
been set aside for evaluation. Most state and local school systems 
operate under an explicit evaluation budget of less than one per 
cent, if that much. The kind of evaluation endeavor we're talking 
about here clearly will demand resources of around five per cent or 
better. The first blush reluctance of individuals to spend that 
kind of money should be countered by hard nosed estimates - of the 
benefits ; both economic and educational, which can be derived from 
rigorously evaluating the progress of a large scale educational 
undertaking. 

Perfection and Imperfectability 

Some detractors will allege quite accurately that systems 
approaches such as those described here are laden with flaws. 

Surely by using objectives-based systems we will discover that 
certain critical features of the educational system are not appraised 
with sufficient sensitivity to yield the right kind of information 
for making proper decisions. When faced with these kinds of criti- 
cisms, however, I am reminded that decisions must currently be made 
regarding educational enterprises, day in and day out. And at the 
moment these decisions are being made with far less sophistication, 
far less data, and far less accuracy than might be rendered under 
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a system such as that proposed herein. Granted that a systems 
analysis approach is not perfect, it nevertheless seems to offer 
a clear improvement over the general quality of decision making 
seen so prevalently these days in educational arenas. And, perhaps 
more importantly, because of its systematic nature such an approach 
is amenable to technical self-correction and, over time, incremental 
improvement so that even if the management system lacks total per- 
fection, it will be so. close that, the learners it serves won't know 

the difference. 
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